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Abstract. We study the typical properties ol polynomial Support Vector Machines 
within a Statistical Mechanics approach that allows us to analyze the effect of 
different normalizations of the features. If the normalization is adecuately chosen, 
there is a hierarchical learning of features of increasing order as a function of the 
training set size. 
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1. Introduction 

The theoretical basis of learning from examples relies on the possibility 
of bounding the generalization error, which is the probability that 
the trained machine makes an error on a new pattern. However, the 
rigorous bounds deduced with methods of statistical theory (Vapnik, 
1995), which hold for the worst case, turn out to be too pessimistic 
in most real world applications. Bounds to the estimator of the gen- 
eralization error, obtained by the leave-one-out technique, are closer 
to experimental results (Vapnik and Chapelle, 2000). This estimator 
is obtained through averaging the classification error on one pattern, 
when learning was achieved by removing that pattern from the training 
set. Another approach to learning theory, heralded more than a decade 
ago by the pioneering work of E. Gardner on perceptrons (Gardner and 
Derrida, 1988), strives to determine analytically the typical properties 
of the learning machine under somewhat restrictive hypothesis. This is 
done using methods from Statistical Physics, developed to study the 
properties of large, disordered physical systems. The basic assumption 
of this approach is that average quantities are representative of the 
machine's properties, with a probability that tends to the unity as the 
system's size goes to infinity, a limit usually called the thermodynamic 
(TD) limit. Thus, in this limit, the leave-one-out estimators and the 
Statistical Mechanics results are expected to converge to the same val- 
ues. The input patterns distribution plays the role of the disorder over 
which averages are taken, the size of the system being the input space 
dimension. In the TD limit, some control parameters, like the training 
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set size relative to the input space dimension, are kept fixed. This 
enables to deduce typical properties for small relative training set sizes, 
in contrast to the bounds provided by the statistical theories, which are 
generally valid for sufficiently large training sets. In this paper we are 
able to get deeper insight on the typical properties of Support Vector 
Machines (SVMs) by also keeping fixed other characteristic quantities. 

The typical properties of polynomial SVMs have been studied in two 
recent papers (Dietrich et al., 1999; Buhot and Gordon, 1999). Both 
consider SVMs in which the input vectors x G 3?" are mapped onto 
quadratic features <& using the normalized mapping (Dietrich et al., 
1999) 

*at(x) = (x, xix/v 7 ^, x 2 x/ v / n, ■ ■ ■ , x n x/^), (1) 
and the non-normalized mapping (Buhot and Gordon, 1999) 

*ATAr(x) = (x,XiX,X 2 X, •••,XfeX). (2) 

respectively. The latter was studied as a function of k, the number of 
quadratic features. For k = n the dimension of both feature spaces is 
the same. They correspond to the quadratic kernels 

K(x,y) = xy (1 + ax-y), (3) 

with a = l/y/n for <J?at and a = 1 for <&nn- In spite of the seemingly 
innocuous differences between the models, their properties are very 
different. 

If the rule to be learned is a linear separation in input space, the gen- 
eralization error e g corresponding to the non-normalized mapping &nn 
is much larger than the Maximal Margin Hyperplane (MMH) solution 
of a simple perceptron in input space (i.e. with no added features). The 
latter corresponds to a linear SVM, which is the smallest SVM able to 
generalize this rule. This difference between the quadratic and the linear 
SVMs increases dramatically with k, the number of quadratic features 
included. On the other hand, the generalization error corresponding to 
the normalized mapping <&n is only slightly larger than that of the 
simple MMH perceptron. In the case of learning quadratic separations, 
the generalization error e g of the normalized mapping exhibits a very 
interesting behaviour (Dietrich et al., 1999; Yoon and Oh, 1998): if 
the number of training patterns scales with n, the dimension of the 
linear subspace, e g decreases up to a finite asymptotic lower bound, 
and it only vanishes asymptotically if the number of patterns scales 
proportionally to n 2 . The generalization error of higher order polyno- 
mial SVMs using the normalized mapping also presents different scaling 
regimes (Dietrich et al., 1999). 
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In order to understand these results, it is useful to consider the 
feature space of the polynomial SVMs as the direct product of several 
subspaces, each one spanned by all the monomials (of the same degree) 
that can be formed with the input components. The number of sub- 
spaces is given by the degree of the polynomial. The dimension of each 
subspace, equal to the number of different monomials, grows polyno- 
mially with n, the input space dimension. For example, in quadratic 
SVM's there are two subspaces, one corresponding to the n linear 
features and the other to the n 2 quadratic features. In the case of the 
normalized mapping, if the training set size scales with the dimension of 
one particular subspace, only the features belonging to this and to the 
lower order subspaces contribute effectively to learning (Dietrich et al., 
1999). As a result, the generalization error decreases asymptotically to 
a lower bound. The latter is smaller the higher the particular subspace 
dimension, and goes to zero when the number of training patterns is 
proportional to the dimension of the highest dimensional subspace. We 
call this behaviour hierarchical learning. 

The only difference between the mappings 3>jv and &nn is that 
the quadratic features in $ are squeezed by a factor a = 1/ ^/n with 
respect to those of &nn- This normalization is very sensitive to the TD 
limit. In particular, at finite n, the hierarchical learning behaviour of 
the generalization error, sharply defined in the TD limit, is expected to 
give raise to crossovers between successive regimes as a function of the 
number of training patterns, in which features belonging to increasingly 
higher order subspaces are learned. 

The aim of the present paper is to clarify to what extent the qualita- 
tive differences between the normalized and the non-normalized map- 
pings are still present in polynomial SVMs working in finite dimension, 
and to characterize the signature of the hierarchical learning. Within 
our approach, given the input space dimension n and the mapping, 
the polynomial SVM is characterized by two kinds of quantities, the 
inflation factor and the feature 's variance in each subspace. The for- 
mer is given by the corresponding subspace dimension relative to the 
input space dimension. The latter is proportional to the normalizing 
factor a; depending on its value, the features distribution may be highly 
anisotropic in the sense that in different subspaces the corresponding 
features have different variances. 

Using the tools of Statistical Mechanics, we determine the typical 
properties of SVMs of finite inflation factors and features variances, as 
a function of a = £/n, where £ is the the number of training patterns. 
This is different from the approaches of (Dietrich et al., 1999; Yoon 
and Oh, 1998) who considered separately two different scalings for I, 
namely i oc n and I oc n 2 . Our results give some insight on the inner 



machrev.tex; 1/02/2008; 14:32; p. 3 



4 



Sebastian Risau-Gusman and Mirta B. Gordon 



workings of finite SVMs, given their inflation factors and variances. The 
different scaling regimes of the generalization error as a function of a 
become a crossover, as expected, but more interestingly, its steepness 
depends not only on the particular SVM considered, but also on the 
complexity of the rule to be learnt. The previous results (Dietrich et 
al., 1999; Buhot and Gordon, 1999) are recovered by performing the 
suitable limits. Our analytical results are supported by the excellent 
agreement with the numerical simulations. 

The paper is organized as follows: in the next section we present 
our model. A short introduction to the Statistical Mechanics approach, 
with its application to our analysis of the quadratic SVMs is presented 
in section 3, the details being left to the Appendix. The analytic pre- 
dictions are described and compared with numerical simulations in 4. 
The results are discussed and generalized in section 5. The conclusion 
is left to section 6. 



We consider the problem of learning a binary classification task from 
examples with a SVM in polynomial feature spaces. The learning set 
contains I patterns (xj, yi) (i = 1, • ■ ■ , £) where Xj is an input vector in 
the n-dimensional input space, and j/j £ { — 1, 1} is its class. We assume 
that the components Xi v [y = l,---,n) are independent identically 
distributed gaussian random variables conveniently standardized, so 
that they have zero-mean and unit variance: 



These input vectors are mapped to a higher dimensional space (the fea- 
ture space) wherein the machine looks for the MMH. In the following we 
concentrate on quadratic feature spaces, although our conclusions are 
more general, and may be applied to higher order polynomial SVMs, as 
discussed in section 5. The mappings ^jyj\f(xj — (x, xix, X2X, • • • , x n x) 
and <J>at(x) = (x, Xlx/^Jn,X2x/^/n, • • • , x n x/ ^/n) are particular in- 
stances of mappings of the form «fr(x) = (0i, • • • , cf> n , (pn, (pu, • • • , 4> n n) 
with (p u = x u , and <p vll = ax u x^. a is the normalizing factor of the 
quadratic components: a = 1 for mapping &nn and a = 1/y/n for <J?jv. 
The patterns probability distribution in feature-space is: 



2. The Model 





-ax v x^). (5) 
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Clearly, the components of are not independent random variables. 
For example, a number 0(n 3 ) of triplets of the form (pu^nr^rv have 
positive correlations. These contribute to the third order moments, 
which should vanish if the features were gaussian. Moreover, the fourth 
order connected correlations (Monasson, 1993) do not vanish in the 
thermodynamic limit. Nevertheless, in the following we will neglect 
these and higher order connected moments. This approximation, used 
in (Buhot and Gordon, 1999) and implicit in (Dietrich et al., 1999), 
is equivalent to assuming that all the components in feature space are 
independent gaussian variables. Then, the only difference between the 
mappings and $>nn lies in the variance of the quadratic compo- 
nents distribution. The results obtained using this simplification are 
in excellent agreement with the numerical tests described in the next 
section. 

Since, due to the symmetry of the transformation, only n(n + l)/2 
among the n 2 quadratic features are different, hereafter we restrict 
the feature space and only consider the non redundant components, 
that we denote z = (z n , z°"). The first n components z u = (z±, ■ ■ ■ , z n ) 
hereafter called u- components, represent the original input pattern of 
unit variance, lying in the linear subspace of the feature space. The 
remaining components z CT = (z n+ \, ■ ■ ■ , Zn) stand for the non redundant 
quadratic features, of variance a, hereafter called a- components, h = 
n(l + A) is the dimension of the restricted feature space, where the 
inflation ratio A is the relative number of non-redundant quadratic 
features per input space dimension. The quadratic mapping has A = 
(n + l)/2. 

According to the preceding discussion, we assume that learning n- 
dimensional patterns selected with the isotropic distribution (4) with 
a quadratic SVM is equivalent to learning the MMH with a simple 
perceptron in an n-dimensional space where the patterns are drawn 
using the following anisotropic distribution, 



P (z) = - — exp 



1 

exp 



2 (2vtct 2 )™ a / 2 



-2^ ' (6) 



where 



A = 2±1, (7) 



no? 



with a the normalization factor in (3). The second moment of the u- 
features is ((z u ) 2 ) = n and that of the a- features is ((z CT ) 2 ) = nAcr 2 . 
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If a 2 A = 1, we get ((z CT ) 2 ) = ((z u ) 2 ), which is the relation satisfied 
by the normalized mapping considered in (Dietrich et al., 1999). The 
non- normalized mapping satisfies a 2 A = n. 



3. Statistical Mechanics 

Statistical Mechanics is the branch of Physics that deals with the prop- 
erties of systems with many degrees of freedom, characterized by their 
energy In our case, this is the cost function E(yf;Ce). It depends on 
the weights w 6 3ft n , whose h = n(l + A) components are the system's 
degrees of freedom, and on the set of training patterns £e, which are 
random variables selected with distribution (6). The equilibrium prop- 
erties at temperature T = [3~ l are deduced through averages over the 
Gibbs distribution: 

^(w; C t ) = exp(-/3£(w; £ e )), (9) 

Zp(£p) is a normalization constant called partition function, defined 
through 

Zf,(£e) = j exp[-/3£(w; £ e )\ p(w) dw, (10) 

where p(w) dw is a measure in the weights' space. When (3 is very large, 
only weights w with energies close to the minimum have significant 
probability, and contribute to the the integral in (10). In the limit (3 — > 
oo, only those that have minimal energy have non- vanishing probability. 

The fraction of training errors corresponding to the minimal cost, 
or any other intensive property of the system, may be deduced from 
the so called free energy //3(A) = — In Zp(£t)/((3n) (or through its 
derivatives with respect to (3), in the limit (3 — ► 00. These quantities 
are random variables, as they depend on the realization of £g. In the 
TD limit n — ► 00, their variance is expected to vanish like 1/y/n, which 
means that all the training sets of the same size are expected to endow 
the system with the same intensive typical properties with probability 
one. This hypothesis, supported in particular by the good agreement 
between the predictions and the simulations on perceptrons, has been 
recently shown to hold on theoretical grounds, at least for some quan- 
tities (Talagrand, 1998). As fp(£i) is equal with probability 1 for all 
the training sets, we can get rid of the dependance on £? by taking 
mean values over all the possible training sets. This non-trivial average 
is done using a sophisticated technique, known as the replica trick, 
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developed for the study of disordered magnetic systems (Mezard et al., 
1987), which uses the identity: 

~Z™ - 1 

InZ = lim (11) 

m— >o m 

where the over line stands for the average over the training sets. The av- 
erage of In Z has been transformed into that of averaging the partition 
functions of m replicated systems. In order to obtain non-trivial results, 
one takes also t — > oo with the number of patterns per input space 
dimension a = £/n constant. If we want to compare the results given 
by this approach with the corresponding quantities of finite machines, 
a is a relevant parameter. 

This formalism has been succesfully applied to the analysis of neural 
networks (Gardner and Derrida, 1988; Seung et al., 1992; Watkin et al., 
1993). In particular, an interesting property is the generalization error 
of the trained machine, which in principle depends on the training set. 
The above assumptions amount to saying that in the TD limit, the 
probability that the generalization error is e g is a delta peak centered 
at the typical value, of zero variance. At finite input dimension n, this 
probability distribution widens, and its mean value is shifted. Both 
these corrections are of order 0{\/\fn). This behaviour, numerically 
verified within several learning scenarios (Buhot et al., 1997; Nadler and 
Fink, 1997; Schroder and Urbanczik, 1998), shows that the predictions 
of the statistical mechanics approach are better for larger n. 

In our case, Cg = {(zj, yi)}i=i — e is the set of training patterns in 
feature space, with = (z",zf) drawn with probability (6), the yi G 
{— 1, +1} being the corresponding classes. The natural cost function E 
is: 

e 

£(w, K ;£ £ ) = ^e( K - 7l ) (12) 

i=l 

where is the Heaviside function, 73 = y^Zj • w/ ^/w • w is the margin 
of pattern Zj, and k is the minimal margin, the smallest allowed dis- 
tance between the hyperplane and the training patterns. Energy (12) 
is thus the number of patterns with margin smaller than n. The per- 
ceptron with weights corresponding to a vanishing cost that maximizes 
k implements the Maximal Margin Hyperplane (MMH). 

The properties of the MMH for the most "natural" pattern distri- 
bution, an isotropic gaussian, have been thoroughly studied (Opper et 
al., 1990; Gordon and Grempel, 1995). The case of a single anisotropy 
axis has also been investigated (Marangi et al., 1995). In our model, 
the properties of a polynomial SVM are those of a MMH perceptron 
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with its inputs lying in the feature space, drawn from the anisotropic 
gaussian (6), with macroscopic subsets of components having differ- 
ent variances. Since the rules to be inferred are assumed to be linear 
separations in feature space, we represent them by the weights w* = 
(w",wj) of a teacher perceptron, so that the class of the patterns is 
y = sign(z • w*). Without any loss of generality we consider normalized 
teachers: w* • w* = n. For the students weights w = (w",w' r ) we take 
the same normalization: w • w = w M • w M + w CT • w' = n . 

Within our model, the values of A and a depend on the particular 
(finite size) SVM we want to analyze, through equations (7) and (8). 
The main difference with respect to previous work (Dietrich et al., 1999; 
Buhot and Gordon, 1999) is that we can make predictions that take into 
account the mapping normalization and some finite-size characteristics 
of the SVM. 

We calculate the typical properties of a particular SVM as follows: 
we fix A and a as explained above, and calculate the averages using 
distributions (9) and (6), in the limits /3^oo,n^oo,^^oo with 
£/n = a constant. A similar procedure has been used in a toy model 
(Risau-Gusman and Gordon, 2000) in the context of Gibbs learning. 
To obtain the properties of the MMH, we look for the maximal value 
of k with vanishing cost (12). The details of the calculations are left to 
the Appendix. In the following we describe the main results. 

It turns out that the typical properties of the SVM can be expressed 
as a function of the (normalized) squared norm of the teacher weights 
in cj-subspace, 

Q* = (13) 

n 

and the following averages (also called order parameters, in physics): 



Q = - z (14) 

n 

RU = \ w (15) 

nV(i-Q)(i-Q*) 



r° = . „Z:\ (is) 

where (• • •) represents the averaging over distribution (9). In the limit 
(3 — > oo, this last averaging is trivial because the MMH is unique. Q is 
the squared norm of the student weights in cr-subspace. R u is the aver- 
age of the overlap between the weights of the teacher and the student 
in the u-subspace. R a is the corresponding overlap in the cr-subspace. 
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Notice that the denominators ensure that both R's are conveniently 
normalized, that is, that their values lie between and 1. 

The generalization error, defined as the probability that the machine 
with weights w makes an error on an unknown pattern, 

E g = J 6 [-(z • w*)(z • w)] P(z) dz, (17) 

can be expressed in terms of the order parameters. After a straightfor- 
ward but cumbersome calculation, we find that its typical value can be 
written as: 



e g = {Eg) = - arccos(i?) (18) 

where 

R = V 19 
V(1 + A-)(1 + AJ) 

with A CT and given by 

A <-> s i^fe- <20> 

The fraction of training patterns that are Support Vectors, averaged 
over all the possible sets of patterns, psv, is a bound to the leave-one- 
out estimator of the generalization error (Vapnik, 1995). Within our 
approach it is straightforward to calculate it: 

psv = 2 H (-tR/VT^R*) Dt. (21) 

where H(x) = Dx, with Dx = exp(-x 2 /2)/v / 2vf. 

Thus, the properties of the SVM depend on the teacher (through 
Q*), on the inflation factor of the feature space A, on the normalizing 
factor a of the mapping (through a) and on the number of patterns per 
input dimension of the training set (a). 



4. Results 

We describe first the experimental data, obtained with quadratic SVMs, 
using both mappings, &nn and <J?jv, which have normalizing factors 
a = 1 and a = l/\/n respectively, where n is the input space dimension. 
The £ = an random input examples of each training set were selected 
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with probability (4) and labelled by teachers of normalized weights 
= (wjj,, w*) drawn at random, are the n components in the linear 
subspace and w* are the n 2 components in the quadratic subspace. We 
do not label the vectors like in the previous section to stress the fact 
that the numerical simulations use the complete feature space, contrary 
to the theoretical approach where we only consider the non-redundant 
components. Notice that, because of the symmetry of the mappings, 
teachers having the same value of the symmetrized weights in the 
quadratic subspace, (wi jl/fJ , + wi :fJ/l/ )/2, are all equivalent. The teachers 
are characterized by the proportion of (squared) weight components in 
the quadratic subspace, Q* = w* • w* /w* • w* . In particular, = and 
Q* = 1 correspond to a purely linear and a purely quadratic teacher 
respectively. The experimental student weights w = (w',w 9 ) were 
obtained by solving numerically the dual problem (Cortes and Vapnik, 
1995), using the Quadratic Optimizer for Pattern Recognition program 
(Vanderbei, 1998; Smola, 1999), that we adapted to the case without 
threshold treated in this paper. We determined Q q = w q • w 9 /n 2 , 
and the normalized overlaps R l = w l ■ w^/a/ (w' • w') (wjj, • w^), and 
R q = w q ■ wi I \J (w 9 • w q ) (w* • w*). They are represented on figures 
1 to 4 as a function of a = l/n, using full and open symbols for 
the mappings and $>nn respectively. Notice that the abscissas 
correspond to the fraction of training patterns per input space dimen- 
sion. For each value of £, averages were performed over a large enough 
number of different teachers and training sets to get error bars smaller 
than the symbols' size in the figures. Experiments were carried out 
for n = 50 and n = 20. The corresponding feature space dimensions 
n(n + 1) are 2550 and 420 respectively. As both show the same trends, 
only results for n = 50 are presented on the figures. For the sake of 
comparison with the theoretical results, we characterize the actual SVM 
by A, its (finite size) inflation factor (7), and a 2 , the variance (8) of 
the components in the cr-subspace. Since n = 50, we have A = 25.5 
and a 2 = 1.960784a 2 , that is a 2 = 1.960784 for the non-normalized 
mapping and a 2 = 0.039216, for the normalized one. 

The theoretical values of Q, the fraction of squared student weights 
in the cr-subspace, and the teacher-student overlaps R u and R a , cor- 
responding to the same classes of teachers as the experimental results 
are represented as lines on the same figures. The excellent agreement 
with the experimental data is striking, and gives an indication that 
the high order correlations of the features, neglected in the model, are 
indeed negligible. Therefore, we have R l = R u , and R q = R a , so that 
from now on we can drop the indices l,q and keep u,a used in the 
theoretical approach. 
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Figure 1. Order parameters of SVMs for purely linear teacher rules, Q* = 0. Symbols 
are experimental results for input space dimension n = 50, corresponding to the two 
kinds of quadratic mappings, $jv with a — (Ml symbols) and <J?jvjv with 

normalizing factor a = 1 (open symbols) respectively. Error bars are smaller than 
the symbols. The lines are not fits, but the solutions of the Statistical Mechanics 
equations for A = (n + l)/2 and a 2 — no 2 /A with n = 50, and a corresponding to 
each mapping. 



Fig. 1 corresponds to a purely linear teacher (Q* = 0), i.e. to a 
quadratic SVM learning a rule linearly separable in input space. As 
in this case R a = because = 0, only R u and Q are repre- 
sented. Conversely, in the case of a purely quadratic rule, = 1, 
represented on fig. 2, R u = 0. Notice that Q —> for a —> oo in 
both cases, meaning that the student learns in which subspace lies 
the teacher. Correspondingly, the normalized teacher-student overlap 
increases smoothly to 1. However, the pace at which these quantities 
reach their asymptotic limit depends on the features normalization 
factor, and it is their combination given by eq. (19) that determines 
the generalization error e g , represented on fig. 5. Clearly, if Q* = 0, 
i.e. if the problem is linearly separable in input space, the normalized 
mapping has lower e g at finite a. Conversely, if the discriminating sur- 
face is purely quadratic in input space, the non-normalized mapping 
gives better results. In the asymptotic limit a — > oo the mapping's 
normalization becomes irrelevant, as in both cases the generalization 
error vanishes. 
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Figure 2. Order parameters of SVMs for purely quadratic teacher rules, Q* = 1. 
Definitions are the same as in figure f . 
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Figure 3. Order parameters of SVMs for isotropic teacher rules, Q» ; iso — A/(l + A). 
Definitions are the same as in figure f . 
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Figure 4- Order parameters of SVMs for a general teacher rule, Q, = 0.5. Definitions 
are the same as in figure 1. 



Fig. 3 shows the results corresponding to the isotropic teacher, = 
Q*-iso = A/(l + A). For A = 25.5 we have Q*-i so = 0.962. A particular 
case of such a teacher, considered in (Dietrich et al., 1999; Yoon and Oh, 
1998), has all its weight components of equal absolute value \w* jV \ = 
\w*,vij\ = 1- Finally, the results corresponding to a general rule, with 

= 0.5, are shown in fig. 4. 

Notice that, irrespective of the mapping, the overlaps R u and R a 
present different behaviours, as the latter increases much slower than 
the former. This reflects the fact that, as the number of quadratic 
components scales like nA, a number of examples of the order of nA are 
needed to learn them. Thus, as a function of a, the linear components 
are learned first. Indeed, R u reaches a value close to 1 with a ~ 0(1) 
while R a needs a ~ O(A) to reach similar values. We call this general 
trend hierarchical learning. 

As the generalization error depends on R u and R a through the com- 
bination (19), the signature of hierarchical learning is present on the 
learning curves e g corresponding to the different rules, plotted against 
a on fig. 5. The performance obtained with the normalized mapping 
is better the smaller the value of Q*. The non- normalized mapping 
shows the opposite trend: its performance for a purely linear teacher 
(Q* = 0) is extremely bad, but it improves for increasing values of 
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Figure 5. Learning curves of SVMs for different teacher rules Q*. Definitions are 
the same as in figure f . The inset is an enlargement of the small a region. 



and slightly overrides that of the normalized mapping in the case of a 
purely quadratic teacher. 

These results reflect the competition on learning the anisotropically 
distributed features. The more the features are compressed, the more 
difficult the learning task. In the case of rules with Q, « 1, the 
linear components carry the most significant information. In those 
cases, it is advantageous to use the normalized mapping, which has 
the cr-components compressed (a 2 = 0.039) with respect to the u- 
components, which have unit variance. 

The non-normalized mapping has a 2 = 1.96, meaning that the 
compressed components are those of the ti-subspace. This mapping is 
better whenever most of the information is contained in the cr-subspace, 
which is the case for teachers with large and, in particular, with 
= 1. In this last case, the linear components only introduce noise 
that hinders the learning process. As the number of linear components 
is much smaller than the number of quadratic ones, their pernicious 
effect is expected to be more conspicuous the smaller the value of A. 

Finally, for the sake of completeness, the fraction of support vectors 
psv = ^sv/^i where £sv is the number of training patterns with max- 
imal stability, is represented on figure 6. Notice that, although these 
curves present qualitatively the same trends as e g , they constitute a 
very loose bound to the latter. Since the student's weights can be 



machrev.tex; 1/02/2008; 14:32; p. 14 



Generalization properties of finite size polynomial Support Vector Machines 15 




expressed as a linear combination of SVs (Vapnik, 1995), this result 
is of practical interest. It shows that increasing the number of training 
patterns does not increase the typical number of non-vanishing coef- 
ficients that have to be determined using the quadratic minimization 
program: although at low a most of the training patterns are expected 
to be support vectors, this fraction decreases smoothly with a. Notice 
that a in our approach is the number of training patterns per input 
space dimension. Thus, even if p$v may seem large, this is not so in 
regard to the dimension of the feature space. In our simulations for 
example, a = 50 is of the order of the VC-dimension of the feature 
space. However, from fig. 6 we expect that less than half of the training 
patterns be SVs. 



5. Discussion 

In order to understand the results obtained in the previous section, we 
first consider the relative behaviour of R u and R a , deduced from the 
theoretical approach. If AJ <C A, which is the case for sufficiently small 
Q*, we get that R a <C R u . This means that the quadratic components 
are more difficult to learn than the linear ones. On the other hand, if the 
teacher lies mainly in the quadratic subspace, A % S> A and R a > R u . 
The crossover between these different behaviours occurs at AJ = A, 
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for which R a = R u . For n = 50, this arises for = 0.998 for the 
normalized mapping and for = 0.929 for the non-normalized one. 
In the particular case of the isotropic teacher and the non-normalized 
mapping, Q* ; j so > 0.929, so that R a > R u , as shown on figure 3. 

These considerations alone are not sufficient to understand the be- 
haviour of the generalization error, which depends on the weighted sum 
of R a and R u (see equation (19)). The behaviour of R a and R u at small 
a is useful to understand the onset of hierarchical learning. In the limit 
a — > 0, we find that Q ~ Aa 2 /(Aa 2 + 1) to leading order in a. This 
result may be understood with the following simple argument: if there 
is only one training pattern, clearly it is a SV and the student's weight 
vector is proportional to it. As a typical example has n components of 
unit length in the u-subspace and nA components of length a in the 
c-subspace, we have Q = nAa 2 / (nAa 2 + n). With the normalized 
mapping, \\m a ^oQ = 1/2. In the case of the non normalized one 
lmio^oQ = (2A — 1)/2A, which depends on the inflation factor of 
the SVM. In this limit, we obtain: 



(22) 



v 7 ! + er 4 A v 7 "' 



R u ~ jl—L—rfi, (23) 




Therefore, R ~ \fa for a <C 1, as for the simple perceptron MMH 
(Gordon and Grempel, 1995), but with a prefactor that depends on 
the mapping and the teacher. 

In our model, we expect that hierarchical learning correspond to a 
fast increase of R at small a, mainly dominated by the contribution of 
R u . As in the limit a —> 0, 



we do not expect to have hierarchical learning if a 4 A S> 1, as in that 
case mainly R a contributes to R. This is what happens with the non 
normalized mapping. On the other hand, hierarchical learning takes 
place if <7 4 A <C 1 and A % < 1. The first condition establishes a con- 
straint on the mapping, which is only satisfied by the normalized one. 
The second condition, that ensures that R a < R u holds, gives the range 
of teachers for which this hierarchical generalization takes place. Under 
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Figure 7. Generalization error of a SVM corresponding to different thermodynamic 
limits. See the text for the definition of a in each regime. 



these conditions, R grows f ast and t he contribution of R a is negligible 
because it is weighted by ct 4 AAJ. The effect of hierarchical learning 
is more important the smaller A£. The most dramatic effect arises for 

= 0, i.e. for a quadratic SVM learning a linearly separable rule. 

Notice that if the normalized mapping is used, the condition AJ < 1 
implies that < Q* ; i so = A/(l + A), where Q*-i so corresponds to 
the isotropic teacher. A straightforward calculation shows that if the 
teachers are drawn at random on the surface of the hypersphere in 
feature space, the distribution of Q* is highly non symmetric, with a 
maximum at the value of Q* ; j so , that depends on n. The fraction of 
teachers with < Q* ; i so is smaller than 1/2. For example, only 47.5% 
of the teachers satisfy this constraint if n = 50. When n — > oo, the 
distribution becomes 5(Q* — Q* ; i S o), and Q*-i so tends to the median, 
meaning that in this limit, only about 50% of the teachers give rise to 
hierarchical learning when using the normalized mapping. 

In the limit a — ► oo, all the generalization error curves converge to 
the same asymptotic value as the simple perceptron MMH learning in 
the feature space, namely e g = 0.500489(1 + A)/a, independently of a 
and Q*. Thus, e g vanishes slower the larger the inflation factor A. 

Since the inflation factor A of the SVM feature space in our approach 
is a free parameter, it does not diverge in the thermodynamic limit 
iV — > oo . As a consequence, the two scaling regimes for e g give rise 
to a simple crossover between a fast decrease at small a followed by a 
slow decrease at large a. The results of Dietrich et al. (Dietrich et al., 
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1999) for the normalized mapping, that corresponds to a 2 A = 1 in our 
model, can be deduced by taking appropriately the limits before solving 
our saddle point equations. The regime where the number of training 
patterns i = an scales with n, is straightforward. It is obtained, within 
our approach, by taking the limit a — > and A — ► oo keeping a 2 A = 1 
and a finite. The regime where the number of training patterns I = 
an scales with nA, the number of quadratic features, is obtained by 
keeping a = a/(l + A) finite whilst taking, here again, the limit a — > 0, 
A — ► oo with a 2 A = 1. The corresponding curves are represented on 
figure 7 for the case of an isotropic teacher. In order to compare with 
our results at finite A, the regime where a is finite is represented as a 
function of a = (1 + A)a using the value of A corresponding to n = 50, 
namely, A = 25.5. In the same figure we represented the generalization 
error e g obtained with our model using the parameter values a 2 = 0.039 
and A = 25.5. 

These results, obtained for quadratic SVMs, are easily generalizable 
to higher order polynomial SVMs, as is the case with the approach of 
(Dietrich et al., 1999). We expect a cascade of hierarchical decreasings 
of the generalization error as a function of a, in which successively more 
and more compressed features are learned. 



6. Conclusion 

We introduced a model that clarifies some aspects of the generaliza- 
tion properties of polynomial Support Vector Machines (SVMs) in 
high dimensional feature spaces. To this end, we focused on quadratic 
SVMs. The quadratic features, which are the pairwise products of input 
components, may be scaled by a normalizing factor. Depending on its 
value, the generalization error presents very different behaviours in the 
thermodynamic limit (Dietrich et al., 1999; Buhot and Gordon, 1999). 

We showed that any finite size quadratic SVM may be characterized 
by two parameters: A and a. The inflation factor A is the number 
of quadratic features relative to the number of input features, and is 
proportional to the input space dimension n. The variance a of the 
quadratic features is related to the normalizing factor. Usually, either 
a ~ 1/v^ (normalized mapping) or a ~ 1 (non normalized mapping). 
In previous studies, not only the input space dimension diverges in the 
thermodynamic limit n — > oo, but also A and a are correspondingly 
scaled. 

In our model, the proportion of quadratic features A and their 
variance a are considered as parameters characterizing the (finite size) 
SVMs. Since we keep them constant when taking the thermodynamic 
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limit, we can study the learning properties of actual SVMs with finite 
inflation ratios and normalizing factors, as a function of a = £/n, where 
t is the number of training examples. 

Our theoretical results were obtained neglecting the correlations 
among the quadratic features. Indeed, their effect does not seem to 
be important, as the agreement between our computer experiments 
with actual SVMs and the theoretical predictions is excellent. This 
approximation was also shown to give good predictions in other sim- 
ilar problems (Dietrich et al., 1999; Yoon and Oh, 1998), but further 
investigations are needed to establish rigorously the conditions of its 
validity. 

We find that the generalization error e g depends on the type of rule 
to be inferred through Q*, the (normalized) teacher's squared weight 
components in the quadratic subspace. If the normalized mapping is 
used and Q* is small enough, the behaviour of e g at small a is domi- 
nated by the high rate learning of the linear components. On increasing 
a, there is a crossover to a regime where the decrease of e g becomes 
much slower. This crossover becomes smoother for increasing values of 
Q*, and this effect of hierarchical learning disappears for large enough 
Q*. If the limits A ~ n — > oo and a 2 ~ 1/n — > are taken together 
with the thermodynamic limit, the hierarchical learning effect gives 
rise to the different scaling regimes, corresponding to t ~ n or t ~ n 2 , 
described previously (Yoon and Oh, 1998; Dietrich et al., 1999). 

On the other hand, if the features are not normalized, the contribu- 
tions of both the linear and the quadratic components to e g are of the 
same order, and there is no hierarchical learning at all. For Q* > Q*- i so , 
which corresponds to the isotropic teacher, the non normalized mapping 
has a slightly smaller generalization error than the normalized one. 

It is worth remarking that if the rule to be learned allows for hierar- 
chical learning, the generalization error of the normalized mapping is 
much smaller than that of the non normalized one. In fact, the teachers 
corresponding to such rules are those with Q* < Q* ; i so , where Q*-i so 
corresponds to the isotropic teacher, the one having all its weights 
components equal. For the others, both the normalized mapping and 
the non normalized one present similar performances. If the weights of 
the teacher are selected at random on a hypersphere in feature space, 
the most probable teachers have precisely = Q*-,iso, and the fraction 
of teachers with < Q* ; j so represent of the order of 50% of the 
inferable rules. Thus, from a practical point of view, without having 
any prior knowledge about the rule underlying a set of examples, the 
normalized mapping should be preferred. 
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Appendix 

The average (11) within the replica approach can be expressed as a 
function of several order parameters, whose values have to minimize 
the free energy of the system. Among these, the overlaps of the weights 
corresponding to different replicas, a, b, 

n 

lab = J2(Wa;vW b;u )/fl, 
v=\ 

h 

Qab = { W a;vW b - v )/fl. (26) 

v=n+l 

As the tasks considered are learnable, the solution that minimizes 
the cost function (12) with maximal k is unique. Thus, in the following 
we may assume that replica symmetry holds. Then, q® b = q u , = 
q a for all a, b, and the saddle point equations corresponding to the 
extremum of the free energy for the MMH are: 



a 



2^Q(1-ct 2 )/ 3 



(R 



a\2 



(i- ( in 2 ) (X+ACT)2 



R° 



1 + A CT ' 
x + A CT 



1 -x 



1- {R a ) 2 \ x + A a 



1-{R U ) 2 ) 1 + A°" ' 

K (R u ? 



\-{R°y a i-(i?«) 2 ' 



(27) 
(28) 
(29) 
(30) 
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R u A 

— = x - (31) 

where A°" and AJ are defined by eq. (20), Q*, Q, R u and R a by- 
equations (13) to (16), and x = lim K —> Kmax (l — q u /(l — Q))/(l — Qa/Q)- 
The integrals in the left hand side of equations (27-29) are 



h = 
h 



+ kH 



/CO 
Dtk(t 



+ R)H 



tR 



(32) 
(33) 
(34) 



with Dt = dt exp (-t 2 /2)/V2^, H(x) = Dt, R is given in eq. (19), 
and 



K = 



V(i-Q)(i + A-) 



(35) 



After solving the above equations for Q, R u , R a , x and k, they allow 
to determine psv and e g through (21) and (18). 
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